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Executive Summary 


Only 60 percent of students in post-secondary education in the United States graduate within six years of 
enrollment. Some minority and disadvantaged students graduate at significantly lower rates. As 
Congress considers reauthorizing the Higher Education Act, some consideration is being given to 
promoting greater use of evidence-based programs and practices to address these gaps. 


One option is to create a tiered evidence grant program to support the development, validation, and 
scaling of evidence-based interventions. Congress has already shown an interest in this approach, with 
the House Education and the Workforce Committee choosing to include a tiered evidence program, called 
Impact Grants, in its reauthorization bill earlier this year. A tiered evidence grant program could help 
address existing college completion disparities for the following reasons: 


e Existing College Completion Rates Are Too Low: The overall college graduation rate is too 
low and even lower for disadvantaged students. The federal government currently spends over 
$26 billion per year for Pell Grants for low-income students, but approximately half of these 
students fail to graduate. The broader economy also suffers as these students fail to become 
prepared for higher-wage jobs. 


e Evidence-based Strategies Can Improve Educational Outcomes: Researchers have begun to 
identify evidence-based interventions — including remedial instructional reforms, academic 
counseling and mentoring, and comprehensive supports — that have been shown to increase 
graduation rates. 


e More Evidence Is Needed of What Works: Most of the existing research on post-secondary 
interventions is still preliminary. Few interventions are backed by strong evidence of 
effectiveness. Funding for further research is small, particularly when compared to the billions of 
dollars spent annually on financial aid and direct assistance to colleges and universities. 


e Tiered Evidence Grants are a Cost-effective Evidence Building Strategy: Tiered evidence 
grants, which have been used successfully in other program areas such as K-12 education, are a 
promising strategy. Because they can be created within existing grant programs, they can build 
the evidence base at little or no extra cost to the federal government. 


This paper reviews the potential benefits of a federal tiered evidence grant program in higher education. It 
includes a review of the existing evidence and draws on lessons learned from other tiered evidence grant 
programs that could help improve its design. 


1 For more information, contact Patrick Lester, Director, Social Innovation Research Center, at (443) 822-4791 or 
patrick@socialinnovationcenter.org. 


Evidence and Post-Secondary Education 


Post-secondary education is a proven pathway to economic opportunity.2 Unfortunately, according to the 
latest data from the U.S. Department of Education, only 59.4 percent of students who enrolled in college 
in 2009 graduated within six years. Moreover, there are significant gaps in the graduation rates among 
different demographic groups. Graduation rates for white students (63.3 percent) and Asian students (73 
percent) are higher than Hispanic (53.6 percent), Native American (41.2 percent), and African American 
(39.5 percent) students.* Completion rates also vary significantly by socio-economic status.® 
Nontraditional older and returning students are also at increased risk of failing to graduate.® 


Closing these gaps, increasing access, and increasing the overall college completion rate are major goals 
for policymakers and post-secondary institutions. However, the data suggest significant variation in the 
effectiveness of schools at achieving these goals. For example, according to data from the Education 
Trust, approximately 40 percent of higher education institutions experience no or small gaps (less than 5 
percent) in the graduation rates between white and Hispanic students, while others have gaps of 20 
percent or more.’ There are similarly wide gaps among institutions for African American students.® 


How can these gaps be closed and the overall graduation rate increased? Over the past two decades 
there has been growing bipartisan interest in using research-based interventions to address such 
disparities. 


For example, the Institute of Education Sciences, which provides federal funding for education research, 
was created during the Bush administration. Several evidence-based initiatives, including the Investing 
in Innovation (i3) program at the Department of Education, were created during the Obama 
administration.‘° More recently, Congress has enacted a number of laws that require the increased use 
of evidence-based programs and practices in K-12 education, school safety, child welfare, and 
employment-related services." 


These efforts have already produced several successes. For example, the i3 program has generated 
positive outcomes across an array of K-12 education programs that were validated by rigorous third-party 
evaluations.'? Two of the program’s largest scale-up grantees, Reading Recovery and KIPP, successfully 


2 Clive Belfield and Thomas Bailey, “The Labor Market Returns to Sub-Baccalaureate College: A Review,” Center for Analysis of 
Postsecondary Education and Employment, March 2017. https://capseecenter.org/labor-market-returns-sub-baccalaureate- 
college-review/; Adam Looney and Constantine Yannelis, "A Crisis in Student Loans? How Changes in the Characteristics of 
Borrowers and in the Institutions They Attended Contributed to Rising Loan Defaults," Brookings Institution, 2015. 
https://www.brookings.edu/wp-content/uploads/2015/09/LooneyTextFall15BPEA. pdf 

3 National Center for Education Statistics, Table 326.10. Available at: 
https://nces.ed.gov/programs/digest/d16/tables/dt16_326.10.asp 

4 Ibid. 

5 Susan Dynarski, "For the Poor, the Graduation Gap is Even Wider than the Enrollment Gap," The New York Times, June 2, 
2015. https://www.nytimes.com/2015/06/02/upshot/for-the-poor-the-graduation-gap-is-even-wider-than-the-enrollment-gap.html; 

8 Jake New, "Repeat Non-Completers," Inside Higher Education, October 7, 2014. 
https://www.insidehighered.com/news/2014/10/07/two-thirds-non-first-time-students-do-not-graduate 

7 The Education Trust, "A Look at Latino Student Success: Identifying Top- and Bottom-Performing Institutions," 2017. 
https://edtrust.org/resource/look-latino-student-success/ 

8 The Education Trust, "A Look at Back Student Success: Identifying Top- and Bottom-Performing Institutions," 2017. 


https://edtrust.org/resource/black-student-success/ 
9 Grover Whitehurst, "Making Education Evidence-Based: Premises, Principles, Pragmatics, and Politics," April 26, 2004. 


https://ies.ed.gov/director/pdf/2004 04 26.pdf 

10 Andrew Feldman and Ron Haskins, "Tiered-Evidence Grantmaking," Evidence-Based Policymaking Collaborative, September 9, 
2016. Available at: http://www.evidencecollaborative.org/toolkits/tiered-evidence-grantmaking; Social Innovation Research 
Center, "Investing in Innovation (i3): Strong Start on Evaluating and Scaling Effective Programs, But Greater Focus Needed on 
Innovation," January 19, 2017. Available at: http://socialinnovationcenter.org/wp-content/uploads/2017/01/SIRC-i3-report.pdf 

| Patrick Lester, "Congress Enacts New Evidence-based School Safety Program,” Social Innovation Research Center, March 23, 
2018. http://www.socialinnovationcenter.org/archives/3288; Patrick Lester, "New Evidence-based Reemployment Services 
Program," Social Innovation Research Center, February 13, 2018. http://www.socialinnovationcenter.org/archives/3271; Patrick 
Lester, "Evidence-based Child Welfare Legislation Enacted," Social Innovation Research Center, February 9, 2018. 
http://www.socialinnovationcenter.org/archives/3153; Patrick Lester, "K-12 Education Bill Advances Evidence-based Policy, 
Replaces i3," Social Innovation Research Center, December 7, 2015. http://www.socialinnovationcenter.org/?p=1806 

2 Patrick Lester, "Investing in Innovation (i3): Strong Start on Evaluating and Scaling Effective Programs, But Greater Focus 
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replicated their programs across dozens of schools.'? These early successes were a precursor to much 
larger scale-up efforts that will begin later this year under the recently enacted Every Student Succeeds 
Act (ESSA)."4 


Could similar results be obtained in post-secondary education? There appears to be growing interest in 
this strategy on Capitol Hill, where both the House and Senate are considering legislation that would 
reauthorize the Higher Education Act.'® 


Earlier this year, the House Education and the Workforce Committee adopted a bill that would create a 
new tiered-evidence initiative, called Impact Grants, in the federal TRIO program.'® The same legislation 
also directs post-secondary institutions to develop evidence-based opioids use prevention programs. 
Given its past support for evidence-related provisions in ESSA, it would not be surprising if the Senate 
also moved legislation with similar provisions. 


The Existing Evidence Base 


As Congress considers legislation that may increase funding for research on, or support greater use of, 

evidence-based practices, it will become increasingly important to understand the state of that evidence. 
While a comprehensive review is beyond the scope of this paper, recent research has identified several 

promising strategies. Examples include: 


e Remedial Instruction Reforms: A large proportion of students enter college unprepared and 
many are assigned to remedial coursework.'’ According to one estimate, approximately half of all 
college students and 70 percent of community college students in 2009 had taken at least one 
remedial course within six months of entering college.'® 


Unfortunately, students who enroll in such courses are less likely to obtain a degree and often 
take longer to graduate.'? Rigorous research on the benefits of remedial instruction has found a 
range of positive, negative, and null effects, although these effects may be more likely to be 
positive for students with greater need.”° 


Research is currently underway to improve these outcomes. Some researchers are testing 
alternative measures to more accurately identify students with greater remediation needs.?' 
Other research is examining alternative or compressed instruction models that may achieve 


Needed on Innovation," Social Innovation Research Center, January 19, 2017. Available at: http://socialinnovationcenter.org/wp- 
content/uploads/2017/01/SIRC-i3-report.pdf 

13 Ibid., p. 45 

14 Patrick Lester, "Evidence-Based Comprehensive School Improvement," Social Innovation Research Center," March 26, 2018. 
http://socialinnovationcenter.org/wp-content/uploads/2018/03/CSI|-turnarounds.pdf 

18 For general background information about the Higher Education Act, see: Alexandra Hegji, "The Higher Education Act (HEA): A 
Primer," Congressional Research Service, August 25, 2017. https://fas.org/sgp/crs/misc/R43351.pdf 

16 The PROSPER Act (HR 4508). See: https://www.congress.gov/bill/115th-congress/house-bill/4508/text. For information about 
the proposed Impact Grants, see Appendix A. 

17 Institute of Education Sciences, "Remedial Coursetaking at U.S. Public 2- and 4-Year Institutions: Scope, Experience, and 
Outcomes," September 2016. https://nces.ed.gov/pubs2016/2016405.pdf 

18 Judith Scott-Clayton, "Evidence-based Reforms in College Remediation are Gaining Steam -- and So Far Living Up to the 


Hype," Brookings Institution, March 29, 2018. https://www.brookings.edu/research/evidence-based-reforms-in-college- 
remediation-are-gaining-steam-and-so-far-living-up-to-the-hype/ 


19 Oliver Schak, et al., "Developmental Education: Challenges and Strategies for Reform," U.S. Department of Education, January 
2017, pp 7-8. https://www2.ed.gov/about/offices/list/opepd/education-strategies. pdf 

20 Judith Scott-Clayton, "Evidence-based Reforms in College Remediation are Gaining Steam -- and So Far Living Up to the 
Hype," Brookings Institution, March 29, 2018. See footnote 7. https://www.brookings.edu/research/evidence-based-reforms-in- 
college-remediation-are-gaining-steam-and-so-far-living-up-to-the-hype/#footnote-7 


21 Community College Research Center, "Improving the Accuracy of Remedial Placement," July 2015. 
https://www.luminafoundation.org/files/resources/improving-accuracy-remedial-placement.pdf; J.J. Hetts and A. Fuenmayor, 
"Promising Pathways to Success: Using Evidence to Dramatically Increase Student Achievement," 2013. 
https://www.accca.org/files/Awards/Mertes%20Award%20-%20LBCC%20Ex%20Summary.pdf 
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better results.22 


e Academic Counseling and Mentoring: Academic counselors who provide advice to 
nontraditional students on issues such as time management and study skills may help them 
achieve better outcomes.?8 Summer counselors who act as mentors in the period immediately 
after high school graduation have been shown to reduce barriers to college enrollment.24 
Mentorships have produced positive results for students with disabilities .26 Some institutions have 
utilized technology-based advisory tools to reduce costs.76 


e Financial Incentives for Student Achievement: Offering monetary awards to students for 
meeting specified academic benchmarks has been found to increase college credits earned and, 
when combined with other supports, produce a small increase in graduation rates.2’ 


e Personalized Learning: Some researchers are testing personalized software or online-based 
coursework to provide tailored instruction.28 


e Early Warning Systems: Some colleges and universities are using predictive analytics and early 
warning systems to identify students who are at risk of failing to graduate and targeting them with 
increased assistance.29 Similar early warning systems have been used in K-12 education.®° 


e Summer Coursework: Studies indicate that students who are enrolled in summer and winter 
coursework earn more academic credits, stay in school longer, and increase their completion 
rates.3! 


e Financial Aid: Personalized text messages have been shown to be effective at encouraging 
students at community colleges to refile their Free Application for Federal Student Aid (FAFSA), 


22 


23 


24 


25 


26 


27 


28 


29 


30 


31 


Elizabeth Zachry Rutschow, et al., "Math in the Real World: Early Findings from a Study of the Dana Center Mathematics 
Pathways," Center for the Analysis of Postsecondary Readiness, May 2017. https://postsecondaryreadiness.org/math-real- 
world-early-findings-dcmp/; Craig Hayward, et al., "Curricular Redesign and Gatekeeper Completion: A Multi-College Evaluation 
of the California Acceleration Project," April 2014. http://cap.3csn.org/files/2014/04/RP-Evaluation-CAP.pdf; Alexandra Logue et 
al., "Reforming Remediation," Education Next, Spring 2017. http://educationnext.org/reforming-remediation-college-students- 
mainstreamed-success-cuny/ 

William N. Evans, et al., "Increasing Community College Completion Rates Among Low-income Students: Evidence from a 
Randomized Controlled Trial Evaluation of a Case Management Intervention," December 2017. 
http://www.nber.org/papers/w24150; Andrew Barr and Benjamin Castleman, "The Bottom Line on College Counseling," 2017. 
http://people.tamu.edu/~abarr/BL_ shell 10 17 2017.pdf; Eric P. Bettinger and Rachel B. Baker, "The Effects of Student 
Coaching: An Evaluation of a Randomized Experiment in Student Advising,” Inside Track, 2013. https://www.insidetrack.com/wp- 
content/uploads/2013/11/aera-educational-evaluation-policy-analysis.pdf; 

What Works Clearinghouse; "Summer Counseling: Transition to College," March 2018. 
https://ies.ed.gov/ncee/wwe/InterventionReport/693 

Sally Lindsay, et al., "A Systematic Review of Mentorship Programs to Facilitate Transition to Post-Secondary Education and 
Employment for Youth and Young Adults with Disabilities," Disability and Rehabilitation, October 2015. 
https://www.tandfonline.com/doi/abs/10.3109/09638288.2015.1092174 

Hoori S. Kalamkarian and Melinda Mechur Karp, "Student Attitudes Toward Technology-mediated Advising Systems," 
Community College Research Center, August 2015. http://ccrc.tc.columbia.edu/media/k2/attachments/student-attitudes-toward- 
technology-mediated-advising-systems.pdf 

Thomas Bailey, et al., "Strategies for Postsecondary Students in Developmental Education — A Practice Guide for College and 
University Administrators, Advisors, and Faculty," Institute of Education Sciences, November 2016, p. 29. 
https://ies.ed.gov/ncee/wwe/PracticeGuide/23 

D. Derek Wu, "Online Learning in Postsecondary Education: A Review of the Empirical Literature," Ithanka S+R, March 11, 
2015. http://www.sr.ithaka.org/wp- 

content/uploads/2015/08/SR_Report Online Learning Postsecondary Education Review Wu_031115.pdf 

MDRC, "Lessons from 15 Years of MDRC’s Postsecondary Research," March 2018. https://www.mdrc.org/publication/lessons- 
15-years-mdrc-s-postsecondary-research; Georgia State University, "Leading With Predictive Analytics," accessed March 30, 
2018, http://success.gsu.edu/approach. 

William Corrin, et al., “Addressing Early Warning Indicators: Interim Impact Findings from the Investing in Innovation (i3) 
Evaluation of Diplomas Now," June 2016. Available at: https://www.mdrc.org/publication/addressing-early-warning-indicators 
Paul Attewell and Sou Hyun Jang, “Summer Coursework and Completing College,” Research in Higher Education, 2013. 
http://aabri.com/manuscripts/131522.pdf; MDRC, "Lessons from 15 Years of MDRC’s Postsecondary Research," March 2018. 


https://www.mdrc.org/publication/lessons-15-years-mdrc-s-postsecondary-research; ; MDRC, "Year-Round Financial Aid," June 
2015. https://www.mdrc.org/publication/year-round-financial-aid 
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thereby helping them to remain enrolled.*? 


e Comprehensive Reforms: Some interventions are more comprehensive in nature, combining 
several strategies at once. One promising example is the Accelerated Study in Associate 
Programs (ASAP) initiative, a pilot program launched by the City University of New York that 
combines developmental courses, tutoring, and financial aid.*? The program doubled the three- 
year graduation rate for enrolled students. It is now being replicated in three other sites.%4 


These are important first steps, but the overall state of the research is still nascent. Many of the 
evaluated projects are relatively new (in some cases, the research is still underway). Many may need 
further refinement. Most have not been widely replicated. 


For example, a 2016 What Works Clearinghouse practice guide on remedial postsecondary education 
reviewed the existing evidence across six related practice areas.*5 Of these, it rated the existing 
evidence as moderate for three and minimal for three. None were rated as having strong evidence of 
effectiveness. 


Another indicator is the number of studies included in the What Works Clearinghouse. In March 2018, it 
included just nine post-secondary interventions, with only five rated as producing positive or potentially 
positive effects.9° By comparison, the clearinghouse included 227 literacy-related interventions with over 
50 rated as producing positive or potentially positive effects.” 


Additional research will require more investment. Federal funding for research on post-secondary access 
and completion comes primarily from the Institute of Education Sciences (IES) at the Department of 
Education and from the National Science Foundation (NSF). In 2016, IES made 12 grants for post- 
secondary and adult education research totaling $20.4 million.38 NSF grants are devoted primarily to 
research on STEM (science, technology, engineering, and math) programs.°9 


Tiered Evidence Grants: Lessons from Past Efforts 


As part of its deliberation on the Higher Education Act, Congress is currently considering ways to further 
build the evidence base in post-secondary education. One option, which has already been advanced in 
the House, is the creation of a tiered-evidence grant initiative.*° 


52 Benjamin Castleman and Lindsay Page, “Freshman Year Financial Aid Nudges: An Experiment to Increase FAFSA Renewal and 
College Persistence,” Journal of Human Resources, 2016.. http://jhr.uwpress.org/content/51/2/389.abstract 

38 Susan Scrivener et al., “Doubling Graduation Rates: Three-Year Effects of CUNY’s Accelerated Study in Associate Programs 
(ASAP) for Developmental Education Students,” MDRC, 2015. htto:/Awww.mdrc.org/project/evaluation-accelerated-study- 
associate-programs-asap-developmental-education-students#overview. 

34 Colleen Sommo and Alyssa Ratledge, "Bringing CUNY Accelerated Study in Associate Programs (ASAP) to Ohio," September 
2016. https://www.mdrc.org/publication/bringing-cuny-accelerated-study-associate-programs-asap-ohio 

35 Thomas Bailey, et al., "Strategies for Postsecondary Students in Developmental Education — A Practice Guide for College and 
University Administrators, Advisors, and Faculty," Institute of Education Sciences, November 2016. 
https://ies.ed.gov/ncee/wwe/PracticeGuide/23 

36 What Works Clearinghouse, “Postsecondary Topic,” accessed March 30, 2018. 
https://ies.ed.gov/ncee/wwc/FWW/Results?filters=,Postsecondary 


3” What Works Clearinghouse, “Literacy Topic,” accessed March 30, 2018. 
https://ies.ed.gov/ncee/wwc/FWW/Results ?filters= Literacy 

38 The National Center for Education Research (NCER) at IES funds two centers devoted to post-secondary and adult education, 
the Center for the Analysis of Postsecondary Readiness (CAPR) and the Center for Analysis of Postsecondary Education and 
Employment (CAPSEE). See: National Center for Education Research: https://ies.ed.gov/ncer/RandD/. For a list of NCER- 
funded research grants and contracts focused on post-secondary and adult education, see: 
https://ies.ed.gov/funding/grantsearch/index.asp?mode=1&sort=5&order=2&searchvals=postsecondary&SearchType=or&checkt 
itle=on&checkaffiliation=on&checkprincipal=on&checkquestion=on&checkprogram=on&checkawardnumber=oné&slctAffiliation=0 


&slctPrincipal=0&slctYear=0&slctProgram=15&sictGoal=0&slctCenter=18&FundType=1&FundType=2 

8° National Science Foundation, Education and Human Resources Research Area: https://www.nsf.gov/dir/index.jsp?org=EHR 

40 Andrew Feldman and Ron Haskins, "Tiered-Evidence Grantmaking," Evidence-Based Policymaking Collaborative, September 9, 
2016. Available at: http://www.evidencecollaborative.org/toolkits/tiered-evidence-grantmaking 
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Tiered evidence initiatives commonly provide two or three levels of grant funding to support both new 
ideas and to validate interventions with some evidence of effectiveness.*1 Smaller developmental grants 
are offered for promising or innovative interventions with only preliminary evidence of effectiveness or that 
are rooted in strong theory. Larger grants are commonly used for interventions with stronger evidence, 
but that need to be validated through replication across multiple sites. The largest scale-up grants are 
usually used to replicate and expand interventions that have already been validated. Such grants 
commonly include cost studies and focus on challenges associated with scaling an evidence-based 
program, including developing appropriate fidelity measures and effective technical assistance. 


The House reauthorization of the Higher Education Act (HR 4508) would create a tiered evidence grant 
program within TRIO, a federal program that supports services for college students from disadvantaged 
backgrounds. The legislation proposes setting aside 10 percent of program funds for a new initiative 
called Impact Grants (see Appendix A).42 As is the case with other tiered evidence initiatives, the 
proposal includes three tiers of grants (early-phase, mid-phase, and expansion). It incorporates evidence 
definitions established under ESSA.*8 It also requires funded interventions to be independently 
evaluated. 


If enacted, the proposed program would receive at least $100 million in funding per year (10 percent of 
TRIO’s funding for FY 2018).44 This would be a significant funding increase for research on post- 
secondary access and completion. 


However, it would also be a small fraction (less than one percent) of the $26.6 billion the federal 
government spent on Pell Grants during the 2016-2017 school year. * About half of these students fail to 
graduate.‘ If a tiered evidence program produced scalable interventions that increased these graduation 
rates, it could more than justify its cost. 


While it is unclear if the House proposal will be enacted as written, there may be similar levels of support 
for evidence-based strategies in both chambers. The same congressional committees with jurisdiction 
over the Higher Education Act also approved the evidence provisions in ESSA in 2015.4” The senior 
Democrat on the Senate HELP Committee, Sen. Patty Murray (D-WA), was also a cosponsor with House 
Speaker Paul Ryan of bipartisan legislation creating a Commission on Evidence-Based Policymaking, 
which reported its findings last year.*® A new tiered evidence grant program in higher education would 
also build on an earlier program, called First in the World, which was briefly funded by Congress from 
2014-2015 (many of these funded projects are still ongoing).*9 


As Congress deliberates on these issues, including the possible creation of a new tiered evidence grant 
program, it may wish to consider lessons learned from other, similar efforts.°° 


41 Andrew Feldman and Ron Haskins, "Tiered-Evidence Grantmaking," Evidence-Based Policymaking Collaborative, September 9, 


2016. Available at: http://www.evidencecollaborative.org/toolkits/tiered-evidence-grantmaking; 

42 The PROSPER Act (HR 4508). See: https://www.congress.gov/bill/115th-congress/house-bill/4508/text 

43 U.S. Department of Education, "Non-Regulatory Guidance: Using Evidence to Strengthen Education Investments," September 
16, 2016. Available at: https://www2.ed.gov/policy/elsec/leg/essa/quidanceuseseinvestment.pdf 

44 Consolidated Appropriations Act, 2018 (H.R. 1625), Division H, p. 136 
http://docs.house.gov/billsthisweek/20180319/DIV%20H%20LABORHHS%20SOM%20FY 18%200MNI.OCR.pdf 

45 Matthew M. Chingos, "Simplifying Grants for College Students: Who Wins and Who Loses?", Brookings Institution, March 8, 
2018. httos://www.brookings.edu/research/simplifying-grants-for-college-students-who-wins-and-who-loses/ 

46 Sarah Butrymowicz, "An Unprecedented Look at Pell Grant Graduation Rates from 1,149 Schools," Hechinger Report, 
September 24, 2015. http://nhechingerreport.org/an-unprecedented-look-at-pell-grant-graduation-rates-from-1149-schools/ 

47 Maggie Severns, "How Congress Finally Killed No Child Left Behind," Politico, December 11, 2015. 
https://www.politico.com/story/2015/12/paul-ryan-congress-no-child-left-behind-216696 

48 Patrick Lester, "Congress Expected to Act on Evidence Commission Recommendations," Social Innovation Research Center, 
September 7, 2017. http://www.socialinnovationcenter.org/?p=2663 

48 See: https://www2.ed.gov/programs/fitw/index.html 

50 Most of these lessons are drawn from the Investing in Innovation program at the U.S. Department of Education (now called the 
Education Innovation and Research program) and the Social Innovation Fund at the Corporation for National and Community 
Service. See: Social Innovation Research Center, "Investing in Innovation (i3): Strong Start on Evaluating and Scaling Effective 
Programs, But Greater Focus Needed on Innovation," January 19, 2017. Available at: http://socialinnovationcenter.org/wp- 
content/uploads/2017/01/SIRC-i3-report.pdf; Social Innovation Research Center, "Social Innovation Fund: Early Results Are 
Promising," June 30, 2015. Available at: http://www.socialinnovationcenter.org/wp- 
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e Targeting Gaps in the Evidence Base: The U.S. Department of Education usually uses 
designated priorities and competitive preferences in its grant programs to steer applicants toward 
areas of interest, consistent with the requirements of the program’s statute.*! In 2017, the 
Department of Education released revised priorities for its discretionary grant programs. 52 These 
priorities would probably need to be further refined for a tiered grant program devoted to college 
access and completion. 


Given the limited resources available for research, the department should focus its priorities on 
identified gaps in the knowledge base. Some federal agencies have established formal learning 
agendas for this purpose.5? Widely utilized interventions that lack sufficient evidence of 
effectiveness are another potential priority. Statutory provisions may help strengthen and 
prioritize such research. 


e Use of High-Capacity Grantees and Experienced Evaluators: In other tiered evidence grant 
programs, success has usually been associated with high-capacity grantees working on projects 
in which both they, and their evaluators, have significant experience.54 Evaluators that have 
previously produced studies that met clearinghouse standards are more likely to do so again in 
the future. 


Atiered evidence grant program should prioritize high-capacity grantees and experienced 
evaluators. In communities where most or all potential grantees are low-capacity, it should 
incentivize the use of high-capacity outside partners. 


e Strong Evaluation Oversight and Technical Assistance: Absent significant oversight and 
technical assistance, evaluations produced by practitioners often fail to meet the high standards 
of rigor that are necessary to be included in an evidence clearinghouse. 


Many (and perhaps most) grantees in the both the i3 program and the Social Innovation Fund at 
the Corporation for National and Community Service began with a poor understanding of the 
programs’ evaluation requirements. To address this issue, both programs quickly established 
extensive oversight mechanisms to ensure that evaluation designs were sufficient to meet 
clearinghouse standards.® Sufficient resources should be set aside to cover the cost of such 
oversight and technical assistance. 


e Early-phase (Innovation) Grants: Innovation is the process of developing and testing new ideas 
that may produce improved outcomes. Innovation often involves high failure rates and can take 
years to produce results.*¢ 


51 
52 


53 


54 


55 


56 


content/uploads/2015/07/Social_Innovation Fund-2015-06-30.pdf 

34 CFR 75.105. See: https://www.law.cornell.edu/cfr/text/34/75.105 

Federal Register, "Education Department: Proposed Supplemental Priorities and Definitions for Discretionary Grant Programs," 
October 12, 2017. Available at: https://www.federalregister.gov/documents/2017/10/12/2017-22127/secretarys-proposed- 
supplemental-priorities-and-definitions-for-disretionary-grant-programs 

The White House, "Analytical Perspectives: Budget of the U.S. Government Fiscal Year 2018," pp. 55-56. Available at: 
https://www.gpo.gov/fdsys/pkg/BUDGET-2018-PER/pdf/BUDGET-2018-PER.pdf. The process of developing a learning agenda 
is described in: USAID Learning Lab, "Learning Agenda," January 25, 2017. Available at: 
https://usaidlearninglab.org/library/learning-agenda 

Social Innovation Research Center, "Social Innovation Fund: Early Results Are Promising,” June 30, 2015, pp. 13-17, 20-22, 36- 
38. Available at: http://www.socialinnovationcenter.org/wp-content/uploads/2015/07/Social_ Innovation Fund-2015-06-30.pdf; 
Social Innovation Research Center, "Investing in Innovation (i3): Strong Start on Evaluating and Scaling Effective Programs, But 
Greater Focus Needed on Innovation," January 19, 2017, pp. 14-16. Available at: http://socialinnovationcenter.org/wp- 
content/uploads/2017/01/SIRC-i3-report.pdf 

Social Innovation Research Center, "Social Innovation Fund: Early Results Are Promising," June 30, 2015, p. 24. Available at: 
http://www.socialinnovationcenter.org/wp-content/uploads/2015/07/Social_ Innovation Fund-2015-06-30.pdf; Social Innovation 
Research Center, "Investing in Innovation (i3): Strong Start on Evaluating and Scaling Effective Programs, But Greater Focus 
Needed on Innovation," January 19, 2017, pp. 29-34. Available at: http://socialinnovationcenter.org/wp- 
content/uploads/2017/01/SIRC-i3-report.pdf 

Studies of new products or strategies by for-profit organizations like Google or Capital One routinely experience failure rates of 
90 percent or more. See Jim Manzi, Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society, 
Basic Books, 2012, pp. 143-167. Similar failure rates are common in health research. 
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Early-stage innovation grants should be smaller, more flexible (allowing rapid testing and 
improvements), allow for planning years or pilot stages, be more tolerant of failure, and allow for 
early termination without penalty if a project is not generating intended results.5” 


e Mid-phase (Validation) Grants: Research has found that even peer-reviewed, published studies 
can experience low replication rates.5® Mid-phase grants are usually intended for interventions 
that have some preliminary evidence (as described above) but need to be validated through 
replication and rigorous impact evaluation across multiple sites. 


Such grants are more likely to produce positive results if they are made to organizations that were 
previously involved in the intervention’s development. Organizations that lack significant previous 
experience with the intervention are more likely to produce null results. 


e Expansion (Scale-up) Grants: Tiered evidence initiatives commonly reserve the largest grants 
for interventions backed by substantial research, often in multi-site studies as described in the 
validation grants above. Scale-up grants should focus on issues that are relevant to successful 
replication, such as the identification and design of fidelity measures, technical assistance for 
partner organizations, cost-related research, and sustainability. 


Such grants often consume a large portion of total funding in a tiered evidence initiative. These 
resources can be stretched further, however, by requiring matching from other public sources, 
including federal funds if allowed under OMB rules or by statute. Requiring public match dollars 
can also infuse evidence into other publicly-funded programs that are not otherwise evidence- 
based. 


Public matching funds can provide a basis for sustainability after the tiered evidence grant ends. 
By contrast, private matching funds, such as those provided by foundations, are usually not 
sustainable. 


e Faster Research: Many of the tiered evidence initiatives developed during the Obama 
administration took a minimum of 4-5 years to complete, and often longer.*? In part, this was 
because these programs encouraged grantees to create new projects in new locations, which 
generated additional planning and set up activities that could delay the beginning of an impact 
evaluation for at least 2-3 years. 


There is a place for longer-term research. By definition, interventions intended to improve 
graduation rates must wait for graduation to occur. However, some interventions and evaluations 
could also focus on shorter-term outcomes, such as year-to-year retention, to provide faster and 
meaningful interim results. Delays could also be reduced by funding projects that are already, to 
the greatest extent possible, fully operational and evaluation-ready. 


Low-cost, short-duration grants are another option. Such evaluations typically must be 
implemented in the first year of a grant and rely heavily on existing administrative data sets to 
lower costs. Examples can be found at the U.S. Department of Education and the Laura and John 
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Social Innovation Research Center, "Investing in Innovation (i3): Strong Start on Evaluating and Scaling Effective Programs, But 
Greater Focus Needed on Innovation," January 19, 2017, pp. 27-29. Available at: http://socialinnovationcenter.org/wp- 
content/uploads/2017/01/SIRC-i3-report.pdf 

Patrick Lester, "Addressing the 'Replication Crisis': evidence-based Policy's Hidden vulnerability," January 19, 2018. 
http://www.socialinnovationcenter.org/?p=2906 

Social Innovation Research Center, "Social Innovation Fund: Early Results Are Promising," June 30, 2015, pp. 11-2. Available at: 
http://www. socialinnovationcenter.org/wp-content/uploads/2015/07/Social_ Innovation Fund-2015-06-30.pdf; Social Innovation 
Research Center, "Investing in Innovation (i3): Strong Start on Evaluating and Scaling Effective Programs, But Greater Focus 
Needed on Innovation," January 19, 2017, pp. 2, 18, 29, 32, 55-56. Available at: http://socialinnovationcenter.org/wp- 
content/uploads/2017/01/SIRC-i3-report.pdf 


Arnold Foundation, both of which have funded such grants.® 


e Improvement-focused Research: Some evidence-based programs may benefit from smaller, 
incremental improvements or adaptations that would make them appropriate for new 
populations.®' High-capacity providers, such as the Nurse-Family Partnership, frequently test 
improvements to their existing evidence-based models. 


Other possible models include improvements tested by the Behavioral Interventions to Advance 
Self-Sufficiency (BIAS) project at the Administration for Children and Families.® Results for 
America has used randomized A/B testing to improve the effectiveness of city services in its What 
Works Cities initiative.6* Rapid-cycle evaluation is a similar promising technique.®° Funding for 
improvement-based research of this kind could be embedded within a low-cost, short-duration 
grant program like the one described above. 


Conclusion 


Increasing college access and closing the existing gaps in graduation rates for disadvantaged and 
minority students are important policy goals. Research has begun to identify several evidence-based 
strategies for addressing these gaps, but more research is needed. 


As Congress considers reauthorizing the Higher Education Act, it should consider creating a tiered 
evidence grant program to support the development, validation, and scaling of evidence-based programs 
in post-secondary education. 
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For an example, see: Administration for Children and Families, "Design Options for an Evaluation of Head Start Coaching: 
Review of Methods for Evaluating Components of Social Interventions," July 2014. Available at: 
https://www.mdrc.org/sites/default/files/nspd task 3 2 review final 11 19 2014new title.pdf 

David Olds, et. al., "Improving the Nurse-Family Partnership in Community Practice," Pediatrics: Volume 132, Supplement 2, 
November 2013. Available at: http://pediatrics.aappublications.org/content/pediatrics/132/Supplement_2/S110.full.pdf 
Administration for Children and Families, "Behavioral Interventions to Advance Self-Sufficiency (BIAS), 2010-2016" at 


https://www.acf.hhs.gov/opre/research/project/behavioral-interventions-to-advance-self-sufficiency; Social Innovation and 
Research Center, "Behavioral Science" at http://www.socialinnovationcenter.org/directory/cross-cutting- 


topics/innovation/behavioral-science 

Patrick Lester, "What Mid-Size Cities Can Teach Feds About Performance," Government Executive, November 18, 2016. 
Available at: http://www.govexec.com/excellence/promising-practices/2016/11/what-mid-size-cities-can-teach-feds-about- 
performance/133300/ 


Mathematica Policy Research, "Rapid-Cycle Evaluation" at: https://www.mathematica-mpr.com/our-capabilities/rapid-cycle- 
evaluation 


Appendix A: Impact Grants Proposal 


The following is the relevant language in HR 4508 for a proposed Impact Grants initiative that would 
be created within the TRIO program.® 


SEC. 4021. IMPACT Grants 


(a) In general_—From funds reserved under subsection (e), the Secretary shall make grants 
to improve postsecondary access and completion rates for qualified individuals from 
disadvantaged backgrounds. These grants shall be known as innovative measures 
promoting postsecondary access and completion grants or IMPACT Grants and allow eligible 
entities to— 


(1) create, develop, implement, replicate, or take to scale evidence-based,*” field- 
initiated innovations, including through pay-for-success initiatives, to serve qualified 
individuals from disadvantaged backgrounds and improve student outcomes; and 


(2) rigorously evaluate such innovations, in accordance with subsection (d). 


(b) Description of grants.—The grants described in subsection (a) shall include— 


(1) early-phase grants to fund the development, implementation, and feasibility 
testing of a program, which prior research suggests has a promise, for the purpose 
of determining whether the program can successfully improve postsecondary access 
and completion rates; 


(2) mid-phase grants to fund implementation and a rigorous evaluation of a program 
that has been successfully implemented under an early-phase grant described in 
paragraph (1); and 


(3) expansion grants to fund implementation and a rigorous replication evaluation of 
a program that has been found to produce sizable, important impacts under a mid- 
phase grant described in paragraph (2) for the purposes of— 

(A) determining whether such outcomes can be successfully reproduced 

and sustained over time; and 

(B) identifying the conditions in which the project is most effective. 


(c) Requirements for Approval of Applications.—To receive a grant under this section, an 
eligible entity shall submit an application to the Secretary at such time, and in such manner 
as the Secretary may require, which shall include— 


(1) an assurance that not less than two-thirds of the individuals who will participate in 
the program proposed to be carried out with the grant will be— 

(A) low-income individuals who are first generation college students; or 

(B) individuals with disabilities; 


(2) an assurance that any other individuals (not described in paragraph (1)) who will 
participate in such proposed program will be— 

(A) low-income individuals; 

(B) first generation college students; or 


The PROSPER Act (HR 4508). See: https://www.congress.gov/bill/115th-congress/house-bill/4508/text 

The term “evidence-based” is defined elsewhere in the bill as having the same meaning as Sec. 8101(21)(A) of the Every 
Student Succeeds Act ((20 U.S.C. 7801(21)(A)). See 
http://uscode.house.gov/view.xhtml?req=(title:20%20section:7801%20edition:prelim) 
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(C) individuals with disabilities; 


(3) a detailed description of the proposed program, including how such program will 
directly benefit students; 


(4) the number of projected students to be served by the program; 

(5) how the program will be evaluated; and 

(6) an assurance that the individuals participating in the project proposed are 

individuals who do not have access to services from another programs funded under 

this section. 
(d) Evaluation.—Each eligible entity receiving a grant under this section shall conduct an 
independent evaluation of the effectiveness of the program carried out with such grant and 
shall submit to the Secretary, on an annual basis, a report that includes— 

(1) a description of how funds received under this section were used; 

(2) the number of students served by the project carried out under this section; and 


(3) a quantitative analysis of the effectiveness of the project. 


(e) Funding.—From amounts appropriated under section 402A(g), the Secretary shall 
reserve not less than 10 percent of such funds to carry out this section. 


About the Social Innovation Research Center: The Social Innovation Research Center (SIRC) is a 
nonpartisan nonprofit research organization focused on social innovation and performance management 
for nonprofits and public agencies. More information about SIRC is available on the organization's web 
site at http:/Awww.socialinnovationcenter.org. 


This paper was developed with the generous financial support of the Laura and John Arnold 
Foundation. The opinions expressed in this paper are the author’s and do not necessarily represent the 
view of the foundation. 
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